Papers with question answering dataset
KnowShiftQA: How Robust are RAG Systems when Textbook Knowledge Shifts in K-12 Education? (2025.acl-short)
Copied to clipboard
| Challenge: | Existing knowledge discrepancies between textbooks and large language models can undermine RAG systems' performance. |
| Approach: | They propose to use a dataset to test RAG system robustness against knowledge discrepancies. |
| Outcome: | The proposed dataset shows that RAG systems suffer performance degradation when faced with knowledge discrepancies. |
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020.tacl-1)
Copied to clipboard
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, Jennimaria Palomaki
| Challenge: | Existing models for multilingual modeling are based on a set of typological features that are used to express meaning in languages such as English. |
| Approach: | They present a question-answer-typed question-referenced dataset that covers 11 typologically diverse languages with 204K question-and-answered pairs. |
| Outcome: | The proposed dataset covers 11 typologically diverse languages with 204K question-answer pairs. |
QUITE: Quantifying Uncertainty in Natural Language Text in Bayesian Reasoning Scenarios (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing probabilistic reasoning datasets require the model to only rank textual alternatives or use limited set of templates. |
| Approach: | They propose a question-answering dataset that uses probabilistic rules to express degrees of certainty. |
| Outcome: | The proposed model outperforms existing models on all reasoning types . it is available on Github and is expected to be used in clinical documentation . |
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (D18-1)
Copied to clipboard
| Challenge: | Existing QA datasets focus on linguistic understanding, but OpenBookQA probes deeper understanding of topic and language. |
| Approach: | They propose a dataset modeled after open book exams for question answering . the open book is a set of 1326 elementary level science facts . human performance on OpenBookQA is close to 92%, they show . |
| Outcome: | The proposed dataset is modeled after open book exams for question answering . human performance on OpenBookQA is close to 92%, but many state-of-the-art QA methods perform poorly . |
Pre-training Universal Language Representation (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the cutting-edge representation learning, most language models focus on specific levels of linguistic units. |
| Approach: | They propose a training objective MiSAD that utilizes meaningful n-grams extracted from large unlabeled corpus by an algorithm for pre-trained language models. |
| Outcome: | The proposed model achieves highest accuracy on analogy tasks in different language levels and significantly improves performance on downstream tasks. |
ComPO: Community Preferences for Language Model Personalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for training language models with human feedback rely on subjective preferences that are assumed to account for an "average" user . however, annotating preferences is inherently subjective and results in generic models that generate outputs not preferred by many user groups. |
| Approach: | They propose a method to personalize preference optimization in LMs by contextualizing the probability distribution of model outputs with the preference provider. |
| Outcome: | The proposed method improves performance by focusing on group-level preferences rather than individual feedback. |